Livious Ear Ie

In this paper we introduce an extension of the Probably Approximately Correct (PAC) learning model to study the problem of learning inclusion hierarchies of concepts (sometimes called is-a hierarchies) from random examples. Using only the hypothesis representations output over many different runs of a learning algorithm, we wish to reconstruct the partial order (with respect to generality) among the different target concepts used to train the algorithm. We give an efficient algorithm for this problem with the property that each run is oblivious of all other runs: each run can take place in isolation, without access to any examples except those of the current target concept, and without access to the current pool of hypothesis representations. Thus, additional mechanisms providing shared information between runs are not necessary for the inference of some nontrivial hierarchies. Introduction and Motivation In this paper we introduce an extension of the Probably Approximately Correct (PAC) learning model (Valiant 1984) to study the problem of learning inclusion hierarchies of concepts (sometimes called is-a hierarchies) from random examples. (In this paper, a concept is simply a set, or equivalently, its Boolean characteristic function.) Using the hypothesis representations output over many different runs of a learning algorithm, we wish to reconstruct the partial order (with respect to generality) among the different target concepts used to train the algorithm. Informally, our intention is to model the ability to not only learn (in the sense of recognition) the abstract concepts chair and furniture, but to also infer that all chairs are furniture. The scenario we have in mind is roughly the following: a learning algorithm L is run many times, and each time is trained on a potentially different target concept f. Each run results in the addition of a new hypothesis representation r to an existing pool. In addition to the usual criterion that each P should provide a good predictive approximation to its corresponding target concept f, we would like the pool of hypothesis representations to effectively support inclusion tests between any pair of target concepts. More precisely, given any ri and r2 in the pool (where ~1 was the result of training L on a target concept fl, and r2 was the result of training L on fz), by examining rl and ~2 we should be able to determine if fi > f2 (that is, f2(4 = 1 implies fi (2) = 1 for all z), fz ,> fr , or fi and fz are incomparable. 1 We will present an algorithm for this problem that meets the following three conditions: 1. 0 blliviousness: Every run of the algorithm can take place in complete isolation, and is oblivious of all past and future runs. When being trained on a target concept f, the algorithm needs access only to examples off. It does not consult the current pool of hypothesis representations or receive examples of other concepts. Thus, all runs of the algorithm are pairwise statistically independent. 2. A Closed System: The pool of hypothesis representations output by the algorithm functions as a closed system. By this we mean that the inclusion test on r1 and r2 takes place without any additional sampling of values of fi or f2, and the information required to determine the relationship between fi and f2 is entirely contained in rl and r2. 3. Succinctness of ypotheses: The learning algorithm does not vacuously satisfy the obliviousness condition by, for instance, storing a large random sample of f as part of its hypothesis representation r and thus letting this random sample become part of the “closed system” of representations. Such an approach would effectively allow each inclusion test to be performed with simultaneous access to examples of both target concepts participating in the test, and would render our first two conditions meaningless. The algorithm explicitly synthesizes from the training data some representation that is considerably more succinct than the training data itself, but that facilitates inclu‘The most natural hypothesis representation T would seem to be simply the representation of some concept h that approximates the corresponding target concept f. While such a representation is clearly sufficient for the standard (predictive) learning problem, we will show that in our model it is not sufficient for supporting inclusion tests, but that allowing t to represent tp~o concepts (that is, 7 = (h, h’), where h and h’ are two different concepts approximating f) is sufficient. We will eventually clarify this issue rigorously, but for now will continue to refer to “hypothesis representations” in an abstract way. Kearns 215 From: AAAI-92 Proceedings. Copyright ©1992, AAAI (www.aaai.org). All rights reserved.